Search Results: "undef"

1 February 2017

Wouter Verhelst: Resetting a Raspberry Pi to default using Ansible

I got myself a bunch of Raspberry Pi 3s a short while ago: Stack of Raspberry Pis ...the idea being that I can use those to run a few experiments on. After the experiment, I would need to be able to somehow remove the stuff that I put on them again. The most obvious way to do that is to reimage them after every experiment; but the one downside of the multi-pi cases that I've put them in is that removing the micro-SD cards from the Pis without removing them from the case, while possible, is somewhat fiddly. That, plus the fact that I cared more about price than about speed when I got the micro-SD cards makes "image all the cards of these Raspberry Pis" something I don't want to do every day. So I needed a different way to reset them to a clean state, and I decided to use ansible to get it done. It required a bit of experimentation, but eventually I came up with this:
---
- name: install package-list fact
  hosts: all
  remote_user: root
  tasks:
  - name: install libjson-perl
    apt:
      pkg: libjson-perl
      state: latest
  - name: create ansible directory
    file:
      path: /etc/ansible
      state: directory
  - name: create facts directory
    file:
      path: /etc/ansible/facts.d
      state: directory
  - name: copy fact into ansible facts directory
    copy:
      dest: /etc/ansible/facts.d/allowclean.fact
      src: allowclean.fact
      mode: 0755
- name: remove extra stuff
  hosts: pis
  remote_user: root
  tasks:
  - apt: pkg=  item   state=absent purge=yes
    with_items: "  ansible_local.allowclean.cleanable_packages  "
- name: remove package facts
  hosts: pis
  remote_user: root
  tasks:
  - file:
      path: /etc/ansible/facts.d
      state: absent
  - apt:
      pkg: libjson-perl
      state: absent
      autoremove: yes
      purge: yes
This ansible playbook has three plays. The first play installs a custom ansible fact (written in perl) that emits a json file which defines cleanable_packages as a list of packages to remove. The second uses that fact to actually remove those packages; and the third removes the fact (and the libjson-perl library we used to produce the JSON). Which means, really, that all the hard work is done inside the allowclean.fact file. That looks like this:
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open PACKAGES, "dpkg -l ";
my @packages;
my %whitelist = (
    ...
);
while(<PACKAGES>)  
    next unless /^ii\s+(\S+)/;
    my $pack = $1;
    next if(exists($whitelist $pack ));
    if($pack =~ /(.*):armhf/)  
        $pack = $1;
     
    push @packages, $1;
 
my %hash;
$hash cleanable_packages  = \@packages;
print to_json(\%hash);
Basically, we create a whitelist, and compare the output of dpkg -l against that whitelist. If a certain package is in the whitelist, then we don't add it to our "packages" list; if it isn't, we do. The whitelist is just a list of package => 1 tuples. We just need the hash keys and don't care about the data, really; I could equally well have made it package => undef, I suppose, but whatever. Creating the whitelist is not that hard. I did it by setting up one raspberry pi to the minimum that I wanted, running
dpkg -l awk '/^ii/ print "\t\""$2"\" => 1," ' > packagelist
and then adding the contents of that packagelist file in place of the ... in the above perl script. Now whenever we apply the ansible playbook above, all packages (except for those in the whitelist) are removed from the system. Doing the same for things like users and files is left as an exercise to the reader. Side note: ... is a valid perl construct. You can write it, and the compiler won't complain. You can't run it, though.

4 January 2017

Rapha&#235;l Hertzog: My Free Software Activities in December 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I was allocated 10 hours to work on security updates for Debian 7 Wheezy. During this time I did the following: Misc packaging With the strong freeze approaching, I had some customer requests to push packages into Debian and/or to fix packages that were in danger of being removed from stretch. While trying to bring back uwsgi into testing I filed #847095 (libmongoclient-dev: Should not conflict with transitional mongodb-dev) and #847207 (uwsgi: FTBFS on multiple architectures with undefined references to uwsgi_* symbols) and interacted on some of the RC bugs that were keeping the package out of testing. I also worked on a few new packages (lua-trink-cjson, lua-inotify, lua-sandbox-extensions) that enhance hindsight in some use cases and sponsored a rozofs update in experimental to fix a file conflict with inn2 (#846571). Misc Debian work Debian Live. I released two live-build updates. The second update added more options to customize the grub configuration (we use it in Kali to override the theme and add more menu entries) both for EFI boot and normal boot. Misc bugreports. #846569 on libsnmp-dev to accomodate the libssl transition (I noticed the package was not maintained, I asked for new maintainers on debian-devel). #847168 on devscripts for debuild that started failing when lintian was failing (unexpected regression). #847318 on lintian to not emit spurious errors for kali packages (which was annoying with the debuild regression above). #847436 for an upgrade problem I got with tryton-server. #847223 on firefoxdriver as it was still depending on iceweasel instead of firefox. Sponsorship. I sponsored a new version of asciidoc (#831965) and of ssldump 0.9b3-6 (for libssl transition). I also uploaded a new version of mutter to fix #846898 (it was ready in SVN already). Distro Tracker Not much happening, I fixed #814315 by switching a few remaining URLs to https. I merged patches from efkin to fix the functional test suite (#814315), that was a really useful contribution! The same contributer started to tackle another ticket (#824912) about adding an API to retrieve action items. This is a larger project and needs some thoughts. I still have to respond to him on his latest patches (after two rounds already). Misc stuff I updated the letsencrypt-sh salt formula for version 0.3.0 and added the possibility to customize the hook script to reload the webserver. The @planetdebian twitter account is no longer working since twitterfeed.com closed doors and the replacement (dlvr.it) is unhappy about the RSS feed of planet.debian.org. I filed bug #848123 against planet-venus since it does not preserve the isPermalink attribute in the guid tag Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

30 December 2016

Chris Lamb: My favourite books of 2016

Whilst I managed to read almost sixty books in 2016 here are ten of my favourites in no particular order. Disappointments this year include Stewart Lee's Content Provider (nothing like his stand-up), Christopher Hitchens' And Yet (his best essays are already published) and Heinlein's Stranger in a Strange Land (great exposition, bizarre conclusion). The worst book I finished, by far, was Mark Edward's Follow You Home.





https://images-eu.ssl-images-amazon.com/images/P/B010EAQLV2.01._PC__.jpg Animal QC Gary Bell, QC Subtitled My Preposterous Life, this rags-to-riches story about a working-class boy turned eminent lawyer would be highly readable as a dry and factual account but I am compelled to include it here for its extremely entertaining style of writing. Full of unsurprising quotes that take one unaware: would you really expect a now-Queen's Counsel to "heartily suggest that if you find yourself suffering from dysentery in foreign climes you do not medicate it with lobster thermidor and a bottle of Ecuadorian red?" A real good yarn.
https://images-eu.ssl-images-amazon.com/images/P/B0196HJ6OS.01._PC__.jpg So You've Been Publically Shamed Jon Ronson The author was initially recommended to me by Brad but I believe I started out with the wrong book. In fact, I even had my doubts about this one, prematurely judging from the title that it was merely cashing-in on a fairly recent internet phenomenon like his more recent shallow take on Trump and the alt-Right but in the end I read Publically Shamed thrice in quick succession. I would particularly endorse the audiobook version: Ronson's deadpan drawl suits his writing perfectly.
https://images-eu.ssl-images-amazon.com/images/P/B00IX49OS4.01._PC__.jpg The Obstacle is the Way Ryan Holiday Whilst everyone else appears to be obligated to include Ryan's recent Ego is the Enemy in their Best of 2016 lists I was actually taken by his earlier "introduction by stealth" to stoic philosophy. Certainly not your typical self-help book, this is "a manual to turn to in troubling times". Returning to this work at least three times over the year even splashing out on the audiobook at some point I feel like I learned a great deal, although it is now difficult to pinpoint exactly what. Perhaps another read in 2017 is thus in order
https://images-eu.ssl-images-amazon.com/images/P/071563335X.01._PC__.jpg Layer Cake J.J. Connolly To judge a book in comparison to the film is to do both a disservice, but reading the book of Layer Cake really underscored just how well the film played to the strengths of that medium. All of the aspects that would not have worked had been carefully excised from the screenplay, ironically leaving more rewarding "layers" for readers attempting the book. A parallel adaption here might be No Country for Old Men - I would love to read (or write) a comparative essay between these two adaptions although McCarthy's novel is certainly the superior source material.
https://images-eu.ssl-images-amazon.com/images/P/B00G1SRB6Q.01._PC__.jpg Lying Sam Harris I've absorbed a lot of Sam Harris's uvre this year in the form of his books but moreover via his compelling podcast. I'm especially fond of Waking Up on spirituality without religion and would rank that as my favourite work of his. Lying is a comparatively short read, more of a long essay in fact, where he argues that we can radically simplify our lives by merely telling the truth in situations where others invariably lie. Whilst it would take a brave soul to adopt his approach his case is superlatively well-argued and a delight to read.
https://images-eu.ssl-images-amazon.com/images/P/0140442103.01._PC__.jpg Letters from a Stoic Seneca

Great pleasure is to be found not only in keeping up an old and established friendship but also in beginning and building up a new one. Reading this in a beautifully svelte hardback, I tackled a randomly-chosen letter per day rather than attempting to read it cover-to-cover. Breaking with a life-long tradition, I even decided to highlight sections in pen so I could return to them at ease. I hope it's not too hackneyed to claim I gained a lot from "building up" a relationship with this book. Alas, it is one of those books that is too easy to recommend given that it might make one appear wise and learned, but if you find yourself in a slump, either in life or in your reading habits, it certainly has my approval.


https://images-eu.ssl-images-amazon.com/images/P/B00BHD3TIE.01._PC__.jpg Solo: A James Bond Novel William Boyd I must have read all of the canonical Fleming novels as a teenager and Solo really rewards anyone who has done so. It would certainly punish anyone expecting a Goldeneye or at least be a little too foreign to be enjoyed. Indeed, its really a pastiche of these originals, both in terms of the time period, general tone (Bond is more somber; more vulnerable) and in various obsessions of Fleming's writing, such as the overly-detailed description of the gambling and dining tables. In this universe, 007's restaurant expenses probably contributed signifcantly to the downfall of the British Empire, let alone his waistline. Bond flicking through a ornithological book at one point was a cute touch
https://images-eu.ssl-images-amazon.com/images/P/B019MMUA8S.01._PC__.jpg The Subtle Art of Not Giving A F*ck Mark Manson Certainly a wildcard to include here and not without its problems, The Subtle Art is a curious manifesto on how to approach life. Whilst Manson expouses an age-old philosophy of grounding yourself and ignoring the accumulation of flatscreen TVs, etc. he manages to do so in a fresh and provocative "21st-centry gonzo" style. Highly entertaining, at one point the author posits an alternative superhero ("Disappointment Panda") that dishes out unsolicited and uncomfortable truths to strangers before simply walking away: "You know, if you make more money, that s not going to make your kids love you," or: "What you consider friendship is really just your constant attempts to impress people." Ouch.
https://images-eu.ssl-images-amazon.com/images/P/B004ZLS5RK.01._PC__.jpg The Fourth Protocol Frederick Forsyth I have a crystal-clear memory from my childhood of watching a single scene from a film in the dead of night: Pierce Brosnan sets a nuclear device to detonate after he can get away but a double-crossing accomplice surreptitiously brings the timetable forward in order that the bomb also disposes of him Anyway, at some point whilst reading The Fourth Protocol it dawned on me that this was that book. I might thus be giving the book more credit due to this highly satisfying connection but I think it stands alone as a superlative political page-turner and is still approachable outside the machinations of the Cold War.
https://images-eu.ssl-images-amazon.com/images/P/B003IDMUSG.01._PC__.jpg The Partner John Grisham After indulging in a bit too much non-fiction and an aborted attempt at The Ministry of Fear, I turned to a few so-called lower-brow writers such as Jeffrey Archer, etc. However, it was The Partner that turned out to be a real page-turner for somewhat undefinable reasons. Alas, it appears the rest of the author's output is unfortunately in the same vein (laywers, etc.) so I am hesitant to immediately begin others but judging from various lists online I am glad I approached this one first.
https://images-eu.ssl-images-amazon.com/images/P/B00D3J2QKC.01._PC__.jpg Shogun: The First Novel of the Asian saga James Clavell Despite its length, I simply couldn't resist returning to Shogun this year although it did fatigue me to the point that I have still yet to commence on its sequel, Tai-Pan. Like any good musical composition, one is always rewarded by returning to a book and I took great delight in uncovering more symbolism throughout (such as noticing that one of the first words Blackthorne learns in Japanese is "truth") but also really savouring the tragic arcs that run throughout the novel, some beautiful phrases ("The day seemed to lose its warmth ") and its wistful themes of inevitability and karma.

30 November 2016

Chris Lamb: Free software activities in November 2016

Here is my monthly update covering what I have been doing in the free software world (previous month):
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced either maliciously or accidentally during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month:

My work in the Reproducible Builds project was also covered in our weekly reports. (#80, #81, #82 #83.

Toolchain issues I submitted the following patches to fix reproducibility-related toolchain issues with Debian:

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.


jenkins.debian.net

jenkins.debian.net runs our comprehensive testing framework.

  • buildinfo.debian.net has moved to SSL. (ac3b9e7)
  • Submit signing keys to keyservers after generation. (bdee6ff)
  • Various cosmetic changes, including
    • Prefer if X not in Y over if not X in Y. (bc23884)
    • No need for a dictionary; let's just use a set. (bf3fb6c)
    • Avoid DRY violation by using a for loop. (4125ec5)

I also submitted 9 patches to fix specific reproducibility issues in apktool, cairo-5c, lava-dispatcher, lava-server, node-rimraf, perlbrew, qsynth, tunnelx & zp.

Debian

Debian LTS This month I have been paid to work 11 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 697-1 for bsdiff fixing an arbitrary write vulnerability.
  • Issued DLA 705-1 for python-imaging correcting a number of memory overflow issues.
  • Issued DLA 713-1 for sniffit where a buffer overflow allowed a specially-crafted configuration file to provide a root shell.
  • Issued DLA 723-1 for libsoap-lite-perl preventing a Billion Laughs XML expansion attack.
  • Issued DLA 724-1 for mcabber fixing a roster push attack.

Uploads
  • redis:
    • 3.2.5-2 Tighten permissions of /var/ lib,log /redis. (#842987)
    • 3.2.5-3 & 3.2.5-4 Improve autopkgtest tests and install upstream's MANIFESTO and README.md documentation.
  • gunicorn (19.6.0-9) Adding autopkgtest tests.
  • libfiu:
    • 0.94-1 Add autopkgtest tests.
    • 0.95-1, 0.95-2 & 0.95-3 New upstream release and improve autopkgtest coverage.
  • python-django (1.10.3-1) New upstream release.
  • aptfs (0.8-3, 0.8-4 & 0.8-5) Adding and subsequently improving the autopkgtext tests.


I performed the following QA uploads:


Finally, I also made the following non-maintainer uploads:
  • libident (0.22-3.1) Move from obsolete Source-Version substvar to binary:Version. (#833195)
  • libpcl1 (1.6-1.1) Move from obsolete Source-Version substvar to binary:Version. (#833196)
  • pygopherd (2.0.18.4+nmu1) Move from obsolete Source-Version substvar to $ source:Version . (#833202)


RC bugs


I also filed 59 FTBFS bugs against arc-gui-clients, asyncpg, blhc, civicrm, d-feet, dpdk, fbpanel, freeciv, freeplane, gant, golang-github-googleapis-gax-go, golang-github-googleapis-proto-client-go, haskell-cabal-install, haskell-fail, haskell-monadcatchio-transformers, hg-git, htsjdk, hyperscan, jasperreports, json-simple, keystone, koji, libapache-mod-musicindex, libcoap, libdr-tarantool-perl, libmath-bigint-gmp-perl, libpng1.6, link-grammar, lua-sql, mediatomb, mitmproxy, ncrack, net-tools, node-dateformat, node-fuzzaldrin-plus, node-nopt, open-infrastructure-system-images, open-infrastructure-system-images, photofloat, ppp, ptlib, python-mpop, python-mysqldb, python-passlib, python-protobix, python-ttystatus, redland, ros-message-generation, ruby-ethon, ruby-nokogiri, salt-formula-ceilometer, spykeviewer, sssd, suil, torus-trooper, trash-cli, twisted-web2, uftp & wide-dhcpv6.

FTP Team

As a Debian FTP assistant I ACCEPTed 70 packages: bbqsql, coz-profiler, cross-toolchain-base, cross-toolchain-base-ports, dgit-test-dummy, django-anymail, django-hstore, django-html-sanitizer, django-impersonate, django-wkhtmltopdf, gcc-6-cross, gcc-defaults, gnome-shell-extension-dashtodock, golang-defaults, golang-github-btcsuite-fastsha256, golang-github-dnephin-cobra, golang-github-docker-go-events, golang-github-gogits-cron, golang-github-opencontainers-image-spec, haskell-debian, kpmcore, libdancer-logger-syslog-perl, libmoox-buildargs-perl, libmoox-role-cloneset-perl, libreoffice, linux-firmware-raspi3, linux-latest, node-babel-runtime, node-big.js, node-buffer-shims, node-charm, node-cliui, node-core-js, node-cpr, node-difflet, node-doctrine, node-duplexer2, node-emojis-list, node-eslint-plugin-flowtype, node-everything.js, node-execa, node-grunt-contrib-coffee, node-grunt-contrib-concat, node-jquery-textcomplete, node-js-tokens, node-json5, node-jsonfile, node-marked-man, node-os-locale, node-sparkles, node-tap-parser, node-time-stamp, node-wrap-ansi, ooniprobe, policycoreutils, pybind11, pygresql, pysynphot, python-axolotl, python-drizzle, python-geoip2, python-mockupdb, python-pyforge, python-sentinels, python-waiting, pythonmagick, r-cran-isocodes, ruby-unicode-display-width, suricata & voctomix-outcasts. I additionally filed 4 RC bugs against packages that had incomplete debian/copyright files against node-cliui, node-core-js, node-cpr & node-grunt-contrib-concat.

21 October 2016

Dirk Eddelbuettel: anytime 0.0.4: New features and fixes

A brand-new release of anytime is now on CRAN following the three earlier releases since mid-September. anytime aims to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects -- and does so without requiring a format string. See the anytime page for a few examples. With release 0.0.4, we add two nice new features. First, NA, NaN and Inf are now simply skipped (similar to what the corresponding Base R functions do). Second, we now also accept large numeric values so that, _e.g., anytime(as.numeric(Sys.time()) also works, effectively adding another input type. We also have squashed an issue reported by the 'undefined behaviour' sanitizer, and the widened the test for when we try to deploy the gettz package get missing timezone information. A quick example of the new features:
anydate(c(NA, NaN, Inf, as.numeric(as.POSIXct("2016-09-01 10:11:12"))))
[1] NA           NA           NA           "2016-09-01"
The NEWS file summarises the release:

Changes in anytime version 0.0.4 (2016-10-20)
  • Before converting via lexical_cast, assign to atomic type via template logic to avoid an UBSAN issue (PR #15 closing issue #14)
  • More robust initialization and timezone information gathering.
  • More robust processing of non-finite input also coping with non-finite values such as NA, NaN and Inf which all return NA
  • Allow numeric POSIXt representation on input, also creating proper POSIXct (or, if requested, Date)

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

31 August 2016

Chris Lamb: Free software activities in August 2016

Here is my monthly update covering what I have been doing in the free software world (previously):

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most Linux distributions provide binary (or "compiled") packages to end users. The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced either maliciously and accidentally during this compilation process by promising identical binary packages are always generated from a given source.

Diffoscope diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them:
  • Added a command-line interface to the try.diffoscope.org web service.
  • Added a JSON comparator.
  • In the HTML output, highlight lines when hovering to make it easier to visually track.
  • Ensure that we pass str types to our Difference class, otherwise we can't be sure we can render them later.
  • Testsuite improvements:
    • Generate test coverage reports.
    • Add tests for Haskell and GitIndex comparators.
    • Completely refactored all of the comparator tests, extracting out commonly-used routines.
    • Confirm rendering of text and HTML presenters when checking non-existing files.
    • Dropped a squashfs test as it was simply too unreliable and/or has too many requirements to satisfy.
  • A large number of miscellaneous cleanups, including:
    • Reworking the comparator setup/preference internals by dynamically importing classes via a single list.
    • Split exceptions out into dedicated diffoscope.exc module.
    • Tidying the PROVIDERS dict in diffoscope/__init__.py.
    • Use html.escape over xml.sax.saxutils.escape, cgi.escape, etc.
    • Removing hard-coding of manual page targets names in debian/rules.
    • Specify all string format arguments as logging function parameters, not using interpolation.
    • Tidying imports, correcting indentation levels and drop unnecessary whitespace.

disorderfs disorderfs is our FUSE filesystem that deliberately introduces nondeterminism in system calls such as readdir(3).
  • Added a testsuite to prevent regressions. (f124965)
  • Added a --sort-dirents=yes no option for forcing deterministic ordering. (2aae325)

Other
  • Improved strip-nondeterminism, our tool to remove specific nondeterministic information after a build:
    • Match more styles of Java .properties files.
    • Remove hyphen from "non-determinism" and "non-deterministic" throughout package for consistency.
  • Improvements to our testing infrastucture:
    • Improve the top-level navigation so that we can always get back to "home" of a package.
    • Give expandable elements cursor: pointer CSS styling to highlight they are clickable.
    • Drop various trailing underlined whitespaces after links.
    • Explicitly log that build was successful or not.
    • Various code-quality improvements, including prefering str.format over concatentation.
  • Miscellaneous updates to our filter-packages internal tool:
    • Add --random=N and --url options.
    • Add support for --show=comments.
    • Correct ordering so that --show-version runs after --filter-ftbfs.
    • Rename --show-ftbfs to --filter-ftbfs and --show-version to --show=version.
  • Created a proof-of-concept reproducible-utils package to contain commonly-used snippets aimed at developers wishing to make their packages reproducible.


I also submitted 92 patches to fix specific reproducibility issues in advi, amora-server, apt-cacher-ng, ara, argyll, audiotools, bam, bedtools, binutils-m68hc1x, botan1.10, broccoli, congress, cookiecutter, dacs, dapl, dateutils, ddd, dicom3tools, dispcalgui, dnssec-trigger, echoping, eekboek, emacspeak, eyed3, fdroidserver, flashrom, fntsample, forkstat, gkrellm, gkrellm, gnunet-gtk, handbrake, hardinfo, ircd-irc2, ircd-ircu, jack-audio-connection-kit, jpy, kxmlgui, libbson, libdc0, libdevel-cover-perl, libfm, libpam-ldap, libquvi, librep, lilyterm, mozvoikko, mp4h, mp4v2, myghty, n2n, nagios-nrpe, nikwi, nmh, nsnake, openhackware, pd-pdstring, phpab, phpdox, phpldapadmin, pixelmed-codec, pleiades, pybit, pygtksourceview, pyicu, python-attrs, python-gflags, quvi, radare2, rc, rest2web, roaraudio, rt-extension-customfieldsonupdate, ruby-compass, ruby-pg, sheepdog, tf5, ttf-tiresias, ttf-tiresias, tuxpaint, tuxpaint-config, twitter-bootstrap3, udpcast, uhub, valknut, varnish, vips, vit, wims, winswitch, wmweather+ & xshisen.

Debian GNU/Linux
Debian LTS

This month I have been paid to work 15 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Authored the patch & issued DLA 596-1 for extplorer, a web-based file manager, fixing an archive traversal exploit.
  • Issued DLA 598-1 for suckless-tools, fixing a segmentation fault in the slock screen locking tool.
  • Issued DLA 599-1 for cracklib2, a pro-active password checker library, fixing a stack-based buffer overflow when parsing large GECOS fields.
  • Improved the find-work internal tool adding optional colour highlighting and migrating it to Python 3.
  • Wrote an lts-missing-uploads tool to find mistakes where there was no correponding package in the archive after an announcement.
  • Added optional colour highlighting to the lts-cve-triage tool.

Uploads
  • redis 2:3.2.3-1 New upstream release, move to the DEP-5 debian/copyright format, ensure that we are running as root in LSB initscripts and add a README.Source regarding our local copies of redis.conf and sentinel.conf.
  • python-django:
    • 1:1.10-1 New upstream release.
    • 1:1.10-2 Fix test failures due to mishandled upstream translation updates.

  • gunicorn:
    • 19.6.0-2 Reload logrotate in the postrotate action to avoid processes writing to the old files and move to DEP-5 debian/copyright format.
    • 19.6.0-3 Drop our /usr/sbin/gunicorn ,3 -debian and related Debian-specific machinery to be more like upstream.
    • 19.6.0-4 Drop "template" systemd .service files and point towards examples and documentation instead.

  • adminer:
    • 4.2.5-1 Take over package maintenance, completely overhauling the packaging with a new upstream version, move to virtual-mysql-server to support MariaDB, updating package names of dependencies and fix the outdated Apache configuration.
    • 4.2.5-2 Correct the php5 package names.




FTP Team As a Debian FTP assistant I ACCEPTed 90 packages: android-platform-external-jsilver, android-platform-frameworks-data-binding, camlpdf, consolation, dfwinreg, diffoscope, django-restricted-resource, django-testproject, django-testscenarios, gitlab-ci-multi-runner, gnome-shell-extension-taskbar, golang-github-flynn-archive-go-shlex, golang-github-jamesclonk-vultr, golang-github-weppos-dnsimple-go, golang-golang-x-time, google-android-ndk-installer, haskell-expiring-cache-map, haskell-hclip, haskell-hdbc-session, haskell-microlens-ghc, haskell-names-th, haskell-persistable-record, haskell-should-not-typecheck, haskell-soap, haskell-soap-tls, haskell-th-reify-compat, haskell-with-location, haskell-wreq, kbtin, libclipboard-perl, libgtk3-simplelist-perl, libjs-jquery-selectize.js, liblemon, libplack-middleware-header-perl, libreoffice, libreswan, libtest-deep-json-perl, libtest-timer-perl, linux, linux-signed, live-tasks, llvm-toolchain-3.8, llvm-toolchain-snapshot, lua-luv, lua-torch-image, lua-torch-nn, magic-wormhole, mini-buildd, ncbi-vdb, node-ast-util, node-es6-module-transpiler, node-es6-promise, node-inline-source-map, node-number-is-nan, node-object-assign, nvidia-graphics-drivers, openhft-chronicle-bytes, openhft-chronicle-core, openhft-chronicle-network, openhft-chronicle-threads, openhft-chronicle-wire, pycodestyle, python-aptly, python-atomicwrites, python-click-log, python-django-casclient, python-git-os-job, python-hypothesis, python-nosehtmloutput, python-overpy, python-parsel, python-prov, python-py, python-schema, python-tackerclient, python-tornado, pyvo, r-cran-cairo, r-cran-mi, r-cran-rcppgsl, r-cran-sem, ruby-curses, ruby-fog-rackspace, ruby-mixlib-archive, ruby-tzinfo-data, salt-formula-swift, scapy3k, self-destructing-cookies, trollius-redis & websploit.

23 August 2016

Reproducible builds folks: Reproducible Builds: week 69 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 14 and Saturday August 20 2016: Fasten your seatbelts Important note: we enabled build path variation for unstable now, so your package(s) might become unreproducible, while previously it was said to be reproducible given a specific build path it probably still is reproducible but read on for the details below in the tests.reproducible-builds.org section! As said many times: this is still research and we are working to make it reality. Media coverage Daniel Stender blogged about python packaging and explained some caveats regarding reproducible builds. Toolchain developments Thomas Schmitt uploaded xorriso which now obeys SOURCE_DATE_EPOCH. As stated in its man pages:
ENVIRONMENT
[...]
SOURCE_DATE_EPOCH  belongs to the specs of reproducible-builds.org.  It
is supposed to be either undefined or to contain a decimal number which
tells the seconds since january 1st 1970. If it contains a number, then
it is used as time value to set the  default  of  --modification-date=,
--gpt_disk_guid,  and  --set_all_file_dates.  Startup files and program
options can override the effect of SOURCE_DATE_EPOCH.
Packages reviewed and fixed, and bugs filed The following packages have become reproducible after being fixed: The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.) The following 2 packages were not changed, but have become reproducible due to changes in their build-dependencies: tagsoup tclx8.4. Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Bug tracker house keeping: Reviews of unreproducible packages 55 package reviews have been added, 161 have been updated and 136 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: Weekly QA work FTBFS bugs have been reported by: diffoscope development Chris Lamb, Holger Levsen and Mattia Rizzolo worked on diffoscope this week. Improvements were made to SquashFS and JSON comparison, the https://try.diffoscope.org/ web service, documentation, packaging, and general code quality. diffoscope 57, 58, and 59 were uploaded to unstable by Chris Lamb. Versions 57 and 58 were both broken, so Holger set up a job on jenkins.debian.net to test diffoscope on each git commit. He also wrote a CONTRIBUTING document to help prevent this from happening in future. From these efforts, we were also able to learn that diffoscope is now reproducible even when built across multiple architectures:
< h01ger>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope.html shows these packages were built on amd64:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb
< h01ger>   and on i386:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb
< h01ger>   and on armhf:
< h01ger>    bd21db708fe91c01ba1c9cb35b9d41a7c9b0db2b 62288 diffoscope_59_all.deb
< h01ger>    366200bf2841136a4c8f8c30bdc87057d59a4cdd 20146 trydiffoscope_59_all.deb
And those also match the binaries uploaded by Chris in his diffoscope 59 binary upload to ftp.debian.org, yay! Eating our own dogfood and enjoying it! tests.reproducible-builds.org Debian related: The last change probably will have an impact you will see: your package might become unreproducible in unstable and this will be shown on tracker.debian.org, while it will still be reproducible in testing. We've done this, because we think reproducible builds are possible with arbitrary build paths. But: we don't think those are a realistic goal for stretch, where we still recommend to use .buildinfo to record the build patch and then do rebuilds using that path. We are doing this, because besides doing theoretical groundwork we also have a practical goal: enable users to independently verify builds. And if they only can do this with a fixed path, so be it. For now :) To be clear: for Stretch we recommend that reproducible builds are done in the same build path as the "original" build. Finally, and just for our future references, when we enabled build path variation on Saturday, August 20th 2016, the numbers for unstable were:
suite all reproducible unreproducible ftbfs depwait not for this arch blacklisted
unstable/amd64 24693 21794 (88.2%) 1753 (7.1%) 972 (3.9%) 65 (0.2%) 95 (0.3%) 10 (0.0%)
unstable/i386 24693 21182 (85.7%) 2349 (9.5%) 972 (3.9%) 76 (0.3%) 103 (0.4%) 10 (0.0%)
unstable/armhf 24693 20889 (84.6%) 2050 (8.3%) 1126 (4.5%) 199 (0.8%) 296 (1.1%) 129 (0.5%)
Misc. Ximin Luo updated our git setup scripts to make it easier for people to write proper descriptions for our repositories. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

24 July 2016

Gregor Herrmann: RC bugs 2016/01-29

seems I've neglected both my blog & my RC bug fixing activities in the last months. anyway, since I still keep track of RC bugs I worked on, I thought I might as well publish the list:

26 June 2016

Clint Adams: A local script for local people

This isn't actually answering the question, but it's close. It's also horrible, so whoever adopts Enrico's script should also completely rewrite this or burn it along with the stack of pizza boxes and the grand piano. Input:
#!/bin/zsh
set -e
PATHS=$(tempfile)
NEWKEYS=$(tempfile)
NEWKEYRING=$(tempfile)
FARTHEST_TEN=$(tempfile)
trap "rm -f $ PATHS  $ NEWKEYS  $ NEWKEYRING  $ FARTHEST_TEN " EXIT
keyring=$ 1:-ksp-dc16.gpg 
myfpr=$ 2:-2100A32C46F895AF3A08783AF6D3495BB0AE9A02 
#keyserver=$ 3:-http://pool.sks-keyservers.net:11371/ 
# this doesn't handle hokey fetch failures
#(for fpr in $(hkt list --keyring $ keyring  --output-format JSON   jq '.[].publickey.fpr')
#do
#  hokey fetch --keyserver "$ keyserver " --validation-method MatchPrimaryKeyFingerprint "$ (Q)fpr "
#done) >$ NEWKEYS 
#
#gpg2 --no-default-keyring --keyring $ NEWKEYRING  --import $ NEWKEYS 
cp "$ keyring " "$ NEWKEYRING "
gpg2 --no-default-keyring --keyring $ NEWKEYRING  --refresh
hkt findpaths --keyring $ NEWKEYRING  '' '' '' > $ PATHS 
id=$(awk -F, "/$ myfpr )\$/  sub(/\(/,BLANKY,\$1);print \$1; " $ PATHS )
grep -e ",\[$ id ," -e ",$ id \]" $ PATHS    sort -n   tail -n 10 > $ FARTHEST_TEN 
targetids=($ (f)"$ $((sed 's/^.*\[//;s/,.*$//;' $ FARTHEST_TEN ; sed 's/\])$//;s/.*,//;' $ FARTHEST_TEN )   sort -n -u   grep -v "^$ id $") " )
targetfprs=($(for i in $ targetids ; do awk -F, "/\($ i ,[^[]/  sub(/\)/,BLANKY,\$2); print \$2 " $ PATHS ; done))
gpg2 --no-default-keyring --keyring $ NEWKEYRING  --list-keys $ targetfprs 
Output:
pub   rsa4096/0x664F1238AA8F138A 2015-07-14 [SC]
      Key fingerprint = 3575 0B8F B6EF 95FF 16B8  EBC0 664F 1238 AA8F 138A
uid                   [ unknown] Daniel Lange <dl.ml1@usrlocal.de>
sub   rsa4096/0x03BEE1C11DB1954B 2015-07-14 [E]
pub   rsa4096/0xDF23DA3396978EB3 2014-09-05 [SC]
      Key fingerprint = BBBC 58B4 5994 CF9C CC56  BCDA DF23 DA33 9697 8EB3
uid                   [  undef ] Michael Meskes <michael@fam-meskes.de>
uid                   [  undef ] Michael Meskes <meskes@postgresql.org>
uid                   [  undef ] Michael Meskes <michael.meskes@credativ.com>
uid                   [  undef ] Michael Meskes <meskes@debian.org>
sub   rsa4096/0x85C3AFFECF0BF9B5 2014-09-05 [E]
sub   rsa4096/0x35D857C0BBCB3B25 2014-11-04 [S]
pub   rsa4096/0x1E953E27D4311E58 2009-07-12 [SC]
      Key fingerprint = C2FE 4BD2 71C1 39B8 6C53  3E46 1E95 3E27 D431 1E58
uid                   [  undef ] Chris Lamb <chris@chris-lamb.co.uk>
uid                   [  undef ] Chris Lamb <lamby@gnu.org>
uid                   [  undef ] Chris Lamb <lamby@debian.org>
sub   rsa4096/0x72B3DBA98575B3F2 2009-07-12 [E]
pub   rsa4096/0xDF6D76C44D696F6B 2014-08-15 [SC] [expires: 2017-06-03]
      Key fingerprint = 1A6F 3E63 9A44 67E8 C347  6525 DF6D 76C4 4D69 6F6B
uid                   [ unknown] Sven Bartscher <sven.bartscher@weltraumschlangen.de>
uid                   [ unknown] Sven Bartscher <svenbartscher@yahoo.de>
uid                   [ unknown] Sven Bartscher <kritzefitz@debian.org>
sub   rsa4096/0x9E83B071ED764C3A 2014-08-15 [E]
sub   rsa4096/0xAEB25323217028C2 2016-06-14 [S]
pub   rsa4096/0x83E33BD7D4DD4CA1 2015-11-12 [SC] [expires: 2017-11-11]
      Key fingerprint = 0B5A 33B8 A26D 6010 9C50  9C6C 83E3 3BD7 D4DD 4CA1
uid                   [ unknown] Jerome Charaoui <jerome@riseup.net>
sub   rsa4096/0x6614611FBD6366E7 2015-11-12 [E]
sub   rsa4096/0xDB17405204ECB364 2015-11-12 [A] [expires: 2017-11-11]
pub   rsa4096/0xF823A2729883C97C 2014-08-26 [SC]
      Key fingerprint = 8ED6 C3F8 BAC9 DB7F C130  A870 F823 A272 9883 C97C
uid                   [ unknown] Lucas Kanashiro <kanashiro@debian.org>
uid                   [ unknown] Lucas Kanashiro <kanashiro.duarte@gmail.com>
sub   rsa4096/0xEE6E5D1A9C2F5EA6 2014-08-26 [E]
pub   rsa4096/0x2EC0FFB3B7301B1F 2014-08-29 [SC] [expires: 2017-04-06]
      Key fingerprint = 76A2 8E42 C981 1D91 E88F  BA5E 2EC0 FFB3 B730 1B1F
uid                   [ unknown] Niko Tyni <ntyni@debian.org>
uid                   [ unknown] Niko Tyni <ntyni@cc.helsinki.fi>
uid                   [ unknown] Niko Tyni <ntyni@iki.fi>
sub   rsa4096/0x129086C411868FD0 2014-08-29 [E] [expires: 2017-04-06]
pub   rsa4096/0xAA761F51CC10C92A 2016-06-20 [SC] [expires: 2018-06-20]
      Key fingerprint = C9DE 2EA8 93EE 4C86 BE73  973A AA76 1F51 CC10 C92A
uid                   [ unknown] Roger Shimizu <rogershimizu@gmail.com>
sub   rsa4096/0x2C2EE1D5DBE7B292 2016-06-20 [E] [expires: 2018-06-20]
sub   rsa4096/0x05C7FD79DD03C4BB 2016-06-20 [S] [expires: 2016-09-18]
Note that this completely neglects potential victims who are unconnected within the KSP set.

2 May 2016

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W17

Last week was quite split into many smaller tasks - working on our libraries (both SQL parser and motranslator got new releases with bug fixes), fixing bugs for upcoming 4.6.1 and working on documentation. From the libraries side, probably most visible is release of motranslator 1.0, just to claim it's now stable enough. Let's see if somebody else will pick it up as well or it will stay only for our use. Most time was however spent on our documentation. We've agreed to move wiki from our server to GitHub wiki and reduce content available on the wiki. So far it's really mixture of user documentation, notes and developer documentation. The final shape should be that wiki will contain only developer documentation and all end user documentation will go to our documentation. So far I've gone through about half of user docs pages, deleted duplicated ones and moved content to our documentation. It is most visible on the user guide which now contains way more information and hopefully it will get more complete in near future. Handled issues:

Filed under: English phpMyAdmin 0 comments

26 April 2016

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W16

Last week was again focused on bug fixing due to increased amount of received bug reports on 4.6.0 release. Fortunately most of the annoying bugs are already fixed in git and will be soon released as 4.6.1. Another bigger task which was started last week was wiki migration. So far we've been using own wiki running MediaWiki and we're migrating it to GitHub wiki. The wiki on GitHub is way simpler, but it seems as better choice for us. During the migration all user documentation will be merged into our documentation, so that it's all in one place and wiki will be targeted on developers. Handled issues:

Filed under: English phpMyAdmin 2 comments

10 April 2016

Vincent Bernat: Testing network software with pytest and Linux namespaces

Started in 2008, lldpd is an implementation of IEEE 802.1AB-2005 (aka LLDP) written in C. While it contains some unit tests, like many other network-related software at the time, the coverage of those is pretty poor: they are hard to write because the code is written in an imperative style and tighly coupled with the system. It would require extensive mocking1. While a rewrite (complete or iterative) would help to make the code more test-friendly, it would be quite an effort and it will likely introduce operational bugs along the way. To get better test coverage, the major features of lldpd are now verified through integration tests. Those tests leverage Linux network namespaces to setup a lightweight and isolated environment for each test. They run through pytest, a powerful testing tool.

pytest in a nutshell pytest is a Python testing tool whose primary use is to write tests for Python applications but is versatile enough for other creative usages. It is bundled with three killer features:
  • you can directly use the assert keyword,
  • you can inject fixtures in any test function, and
  • you can parametrize tests.

Assertions With unittest, the unit testing framework included with Python, and many similar frameworks, unit tests have to be encapsulated into a class and use the provided assertion methods. For example:
class testArithmetics(unittest.TestCase):
    def test_addition(self):
        self.assertEqual(1 + 3, 4)
The equivalent with pytest is simpler and more readable:
def test_addition():
    assert 1 + 3 == 4
pytest will analyze the AST and display useful error messages in case of failure. For further information, see Benjamin Peterson s article.

Fixtures A fixture is the set of actions performed in order to prepare the system to run some tests. With classic frameworks, you can only define one fixture for a set of tests:
class testInVM(unittest.TestCase):
    def setUp(self):
        self.vm = VM('Test-VM')
        self.vm.start()
        self.ssh = SSHClient()
        self.ssh.connect(self.vm.public_ip)
    def tearDown(self):
        self.ssh.close()
        self.vm.destroy()
    def test_hello(self):
        stdin, stdout, stderr = self.ssh.exec_command("echo hello")
        stdin.close()
        self.assertEqual(stderr.read(), b"")
        self.assertEqual(stdout.read(), b"hello\n")
In the example above, we want to test various commands on a remote VM. The fixture launches a new VM and configure an SSH connection. However, if the SSH connection cannot be established, the fixture will fail and the tearDown() method won t be invoked. The VM will be left running. Instead, with pytest, we could do this:
@pytest.yield_fixture
def vm():
    r = VM('Test-VM')
    r.start()
    yield r
    r.destroy()
@pytest.yield_fixture
def ssh(vm):
    ssh = SSHClient()
    ssh.connect(vm.public_ip)
    yield ssh
    ssh.close()
def test_hello(ssh):
    stdin, stdout, stderr = ssh.exec_command("echo hello")
    stdin.close()
    stderr.read() == b""
    stdout.read() == b"hello\n"
The first fixture will provide a freshly booted VM. The second one will setup an SSH connection to the VM provided as an argument. Fixtures are used through dependency injection: just give their names in the signature of the test functions and fixtures that need them. Each fixture only handle the lifetime of one entity. Whatever a dependent test function or fixture succeeds or fails, the VM will always be finally destroyed.

Parameters If you want to run the same test several times with a varying parameter, you can dynamically create test functions or use one test function with a loop. With pytest, you can parametrize test functions and fixtures:
@pytest.mark.parametrize("n1, n2, expected", [
    (1, 3, 4),
    (8, 20, 28),
    (-4, 0, -4)])
def test_addition(n1, n2, expected):
    assert n1 + n2 == expected

Testing lldpd The general plan for to test a feature in lldpd is the following:
  1. Setup two namespaces.
  2. Create a virtual link between them.
  3. Spawn a lldpd process in each namespace.
  4. Test the feature in one namespace.
  5. Check with lldpcli we get the expected result in the other.
Here is a typical test using the most interesting features of pytest:
@pytest.mark.skipif('LLDP-MED' not in pytest.config.lldpd.features,
                    reason="LLDP-MED not supported")
@pytest.mark.parametrize("classe, expected", [
    (1, "Generic Endpoint (Class I)"),
    (2, "Media Endpoint (Class II)"),
    (3, "Communication Device Endpoint (Class III)"),
    (4, "Network Connectivity Device")])
def test_med_devicetype(lldpd, lldpcli, namespaces, links,
                        classe, expected):
    links(namespaces(1), namespaces(2))
    with namespaces(1):
        lldpd("-r")
    with namespaces(2):
        lldpd("-M", str(classe))
    with namespaces(1):
        out = lldpcli("-f", "keyvalue", "show", "neighbors", "details")
        assert out['lldp.eth0.lldp-med.device-type'] == expected
First, the test will be executed only if lldpd was compiled with LLDP-MED support. Second, the test is parametrized. We will execute four distinct tests, one for each role that lldpd should be able to take as an LLDP-MED-enabled endpoint. The signature of the test has four parameters that are not covered by the parametrize() decorator: lldpd, lldpcli, namespaces and links. They are fixtures. A lot of magic happen in those to keep the actual tests short:
  • lldpd is a factory to spawn an instance of lldpd. When called, it will setup the current namespace (setting up the chroot, creating the user and group for privilege separation, replacing some files to be distribution-agnostic, ), then call lldpd with the additional parameters provided. The output is recorded and added to the test report in case of failure. The module also contains the creation of the pytest.config.lldpd object that is used to record the features supported by lldpd and skip non-matching tests. You can read fixtures/programs.py for more details.
  • lldpcli is also a factory, but it spawns instances of lldpcli, the client to query lldpd. Moreover, it will parse the output in a dictionary to reduce boilerplate.
  • namespaces is one of the most interesting pieces. It is a factory for Linux namespaces. It will spawn a new namespace or refer to an existing one. It is possible to switch from one namespace to another (with with) as they are contexts. Behind the scene, the factory maintains the appropriate file descriptors for each namespace and switch to them with setns(). Once the test is done, everything is wipped out as the file descriptors are garbage collected. You can read fixtures/namespaces.py for more details. It is quite reusable in other projects2.
  • links contains helpers to handle network interfaces: creation of virtual ethernet link between namespaces, creation of bridges, bonds and VLAN, etc. It relies on the pyroute2 module. You can read fixtures/network.py for more details.
You can see an example of a test run on the Travis build for 0.9.2. Since each test is correctly isolated, it s possible to run parallel tests with pytest -n 10 --boxed. To catch even more bugs, both the address sanitizer (ASAN) and the undefined behavior sanitizer (UBSAN) are enabled. In case of a problem, notably a memory leak, the faulty program will exit with a non-zero exit code and the associated test will fail.

  1. A project like cwrap would definitely help. However, it lacks support for Netlink and raw sockets that are essential in lldpd operations.
  2. There are three main limitations in the use of namespaces with this fixture. First, when creating a user namespace, only root is mapped to the current user. With lldpd, we have two users (root and _lldpd). Therefore, the tests have to run as root. The second limitation is with the PID namespace. It s not possible for a process to switch from one PID namespace to another. When you call setns() on a PID namespace, only children of the current process will be in the new PID namespace. The PID namespace is convenient to ensure everyone gets killed once the tests are terminated but you must keep in mind that /proc must be mounted in children only. The third limitation is that, for some namespaces (PID and user), all threads of a process must be part of the same namespace. Therefore, don t use threads in tests. Use multiprocessing module instead.

4 March 2016

Antonio Terceiro: Debian Ruby Sprint 2016 - day 4: Steady Progress, Deferred Spring Cleaning, and Capital Sins

As the day 4 of the Debian Ruby team sprint in Curitiba unfolded, we have now fixed a total of more than 70 build failure bugs, managed to almost finish the Ruby 2.3 transition to be good to migrate into testing, and bootstrapped some documentation that will help new contributors get up to speed with Ruby packaging faster. We have also requested the removal of several packages that are either severely outdated, abandoned upstream, beyond repair, utterly wrong, or in some cases, all of the above. The full list of work items finished yesterday is: We also managed to flirt with 2 capital sins. For those who care about these things, which I don t (but I still care about you), I guess 2 out of 7 still means we are good? :-) I few people that I will not name complained that they hadn t had enough steak on the previous night, so we set out to visit a traditional all-you-can-eat Brazilian steakhouse ( churrascaria ). I made a reservation at Jardins Grill and there you have gluttony. I am pretty sure that not enough steak wasn t an issue last night. You can see how happy, despite being full to almost the point of being sick, everyone was. A disjunct set of people, who I will also not name, were very disappointed to find out that the ruby-tinder package has absolutely nothing to do with Tinder but were still very active on the later. Maybe Friday night we will have to split the group into a lust-free family party and a Tinder party.

16 February 2016

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W06

As we're getting closer to release of new major release (4.6.0), the focus moves to the bug fixing. Most of the fixed issues came from our error reporting server, which collects error reports from the installations. Vast majority if bugs were affecting older releases as well, so these were fixed for upcoming 4.5.5 as well. Besides bugfixing there was also some cleanups in the master branch - shared code for processing sprites and covering it by tests and removed caching of MySQL server information as that didn't bring any speed improvements and lead to cache consistency issues. All handled issues:

Filed under: English phpMyAdmin 0 comments

2 February 2016

Sune Vuorela: Compilers and error messages

So. I typo ed up some template code the other day. And once again I learned the importance of using several c++ compilers. Here is a very reduced version of my code:
#include <utility>
template <typename T> auto foo(const T& t) -> decltype(x.first)

return t.first;

int main()

foo(std::make_pair(1,2));
return 0;
And let s start with the compiler I was testing with first. MSVC (2013 and 2015)
main.cpp(8): error C2672: foo : no matching overloaded function found
main.cpp(8): error C2893: Failed to specialize function template unknown-type foo(const T &)
It is not completely clear from that error message what s going on, so let s try some other compilers: GCC (4.9-5.3)
2 : error: x was not declared in this scope
template <typename T> auto foo(const T& t) -> decltype(x.first)
That s pretty clear. More compilers: Clang (3.3-3.7)
2 : error: use of undeclared identifier x
template <typename T> auto foo(const T& t) -> decltype(x.first)
ICC (13)
example.cpp(2): error: identifier x is undefined
template <typename T> auto foo(const T& t) -> decltype(x.first)
(Yes. I mistyped the variable name used for decltype. Replacing the x with t makes it build). Thanks to http://gcc.godbolt.org/ and http://webcompiler.cloudapp.net/ for testing with various compilers.

22 December 2015

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions

Again one week has passed and there has been some progress on phpMyAdmin. This week the focus was on code cleanup and issue fixing. I've identified several pieces of dead code and removed that from our code base (the biggest removal being no longer used PMA_String class). The codebase is quite old and sometimes it's not easy to identify code which is not really being used, as it still might be referenced from some locations, but without actually invoking it. On the issue fixing side, I'm really happy from improving user experience with fields marked as binary - so far phpMyAdmin has displayed them as hexadecimal dump in all cases. Now it will display them as text as long as the content is valid utf-8. Bugs fixed: PS: I will most likely skip report next week due to Christmas.

Filed under: English phpMyAdmin 0 comments

13 December 2015

Gregor Herrmann: RC bugs 2015/38-50

it looks like this autumn was not my best blogging time: this is the first posting in 3 months. anyway, I wanted to give a quick overview about my work on RC bugs. again nothing exciting, mostly just trying to fix the ones popping up in the pkg-perl team.

25 September 2015

Dirk Eddelbuettel: RcppEigen 0.3.2.5.1

A bugfix release of RcppEigen is now on CRAN and in Debian. The NEWS file entry follows.
Changes in RcppEigen version 0.3.2.5.1 (2015-09-23)
  • Corrected use of kitten() thanks to Grant Brown (#21)
  • Applied upstream change to protect against undefined behaviour with null pointers
Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 August 2015

Joey Hess: then and now

It's 2004 and I'm in Oldenburg DE, working on the Debian Installer. Colin and I pair program on partman, its new partitioner, to get it into shape. We've somewhat reluctantly decided to use it. Partman is in some ways a beautful piece of work, a mass of semi-object-oriented, super extensible shell code that sprang fully formed from the brow of Anton. And in many ways, it's mad, full of sector alignment twiddling math implemented in tens of thousands of lines of shell script scattered amoung hundreds of tiny files that are impossible to keep straight. In the tiny Oldenburg Developers Meeting, full of obscure hardware and crazy intensity of ideas like porting Debian to VAXen, we hack late into the night, night after night, and crash on the floor. sepia toned hackers round a table It's 2015 and I'm at a Chinese bakery, then at the Berkeley pier, then in a SF food truck lot, catching half an hour here and there in my vacation to add some features to Propellor. Mostly writing down data types for things like filesystem formats, partition layouts, and then some small amount of haskell code to use them in generic ways. Putting these peices together and reusing stuff already in Propellor (like chroot creation). Before long I have this, which is only 2 undefined functions away from (probably) working:
let chroot d = Chroot.debootstrapped (System (Debian Unstable) "amd64") mempty d
        & Apt.installed ["openssh-server"]
        & ...
    partitions = fitChrootSize MSDOS
        [ (Just "/boot", mkPartiton EXT2)
        , (Just "/", mkPartition EXT4)
        , (Nothing, const (mkPartition LinuxSwap (MegaBytes 256)))
        ]
 in Diskimage.built chroot partitions (grubBooted PC)
This is at least a replication of vmdebootstrap, generating a bootable disk image from that config and 400 lines of code, with enormous customizability of the disk image contents, using all the abilities of Propellor. But is also, effectively, a replication of everything partman is used for (aside from UI and RAID/LVM). sailboat on the SF bay What a difference a decade and better choices of architecture make! In many ways, this is the loosely coupled, extensible, highly configurable system partman aspired to be. Plus elegance. And I'm writing it on a lark, because I have some spare half hours in my vacation. Past Debian Installer team lead Tollef stops by for lunch, I show him the code, and we have the conversation old d-i developers always have about partman. I can't say that partman was a failure, because it's been used by millions to install Debian and Ubuntu and etc for a decade. Anything that deletes that many Windows partitions is a success. But it's been an unhappy success. Nobody has ever had a good time writing partman recipes; the code has grown duplication and unmaintainability. I can't say that these extensions to Propellor will be a success; there's no plan here to replace Debian Installer (although with a few hundred more lines of code, propellor is d-i 2.0); indeed I'm just adding generic useful stuff and building further stuff out of it without any particular end goal. Perhaps that's the real difference.

29 June 2015

Lunar: Reproducible builds: week 9 in Stretch cycle

What happened about the reproducible builds effort this week: Toolchain fixes Norbert Preining uploaded texinfo/6.0.0.dfsg.1-2 which makes texinfo indices reproducible. Original patch by Chris Lamb. Lunar submitted recently rebased patches to make the file order of files inside .deb stable. akira filled #789843 to make tex4ht stop printing timestamps in its HTML output by default. Dhole wrote a patch for xutils-dev to prevent timestamps when creating gzip compresed files. Reiner Herrmann sent a follow-up patch for wheel to use UTC as timezone when outputing timestamps. Mattia Rizzolo started a discussion regarding the failure to build from source of subversion when -Wdate-time is added to CPPFLAGS which happens when asking dpkg-buildflags to use the reproducible profile. SWIG errors out because it doesn't recognize the aforementioned flag. Trying to get the .buildinfo specification to more definitive state, Lunar started a discussion on storing the checksums of the binary package used in dpkg status database. akira discovered while proposing a fix for simgrid that CMake internal command to create tarballs would record a timestamp in the gzip header. A way to prevent it is to use the GZIP environment variable to ask gzip not to store timestamps, but this will soon become unsupported. It's up for discussion if the best place to fix the problem would be to fix it for all CMake users at once. Infrastructure-related work Andreas Henriksson did a delayed NMU upload of pbuilder which adds minimal support for build profiles and includes several fixes from Mattia Rizzolo affecting reproducibility tests. Neils Thykier uploaded lintian which both raises the severity of package-contains-timestamped-gzip and avoids false positives for this tag (thanks to Tomasz Buchert). Petter Reinholdtsen filled #789761 suggesting that how-can-i-help should prompt its users about fixing reproducibility issues. Packages fixed The following packages became reproducible due to changes in their build dependencies: autorun4linuxcd, libwildmagic, lifelines, plexus-i18n, texlive-base, texlive-extra, texlive-lang. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Untested uploaded as they are not in main: Patches submitted which have not made their way to the archive yet: debbindiff development debbindiff/23 includes a few bugfixes by Helmut Grohne that result in a significant speedup (especially on larger files). It used to exhibit the quadratic time string concatenation antipattern. Version 24 was released on June 23rd in a hurry to fix an undefined variable introduced in the previous version. (Reiner Herrmann) debbindiff now has a test suite! It is written using the PyTest framework (thanks Isis Lovecruft for the suggestion). The current focus has been on the comparators, and we are now at 93% of code coverage for these modules. Several problems were identified and fixed in the process: paths appearing in output of javap, readelf, objdump, zipinfo, unsqusahfs; useless MD5 checksum and last modified date in javap output; bad handling of charsets in PO files; the destination path for gzip compressed files not ending in .gz; only metadata of cpio archives were actually compared. stat output was further trimmed to make directory comparison more useful. Having the test suite enabled a refactoring of how comparators were written, switching from a forest of differences to a single tree. This helped removing dust from the oldest parts of the code. Together with some other small changes, version 25 was released on June 27th. A follow up release was made the next day to fix a hole in the test suite and the resulting unidentified leftover from the comparator refactoring. (Lunar) Documentation update Ximin Luo improved code examples for some proposed environment variables for reference timestamps. Dhole added an example on how to fix timestamps C pre-processor macros by adding a way to set the build date externally. akira documented her fix for tex4ht timestamps. Package reviews 94 obsolete reviews have been removed, 330 added and 153 updated this week. Hats off for Chris West (Faux) who investigated many fail to build from source issues and reported the relevant bugs. Slight improvements were made to the scripts for editing the review database, edit-notes and clean-notes. (Mattia Rizzolo) Meetings A meeting was held on June 23rd. Minutes are available. The next meeting will happen on Tuesday 2015-07-07 at 17:00 UTC. Misc. The Linux Foundation announced that it was funding the work of Lunar and h01ger on reproducible builds in Debian and other distributions. This was further relayed in a Bits from Debian blog post.

Next.

Previous.